(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



'""■"'SSK 0 *** fISgy nminiu «n in urn 

(43) International Publication Date (10) International Publication Number 

7 March 2002 (07.03.2002) PCT WO 02/19698 A2 

(51) International Patent Classification 7 : H04N 5/00 (74) Agents: MORNEAULT, Monique, A. et al.; Wallenstein 

& Wagner, Ltd., 311 South Wacker Drive - 5300, Chicago, 



(21) International Application Number: PCT/US0 1/27351 



IL 60606 (US). 



(22) International Filing Date: 31 August 2001 (31.08.2001) 



(25) Filing Language: English 

(26) Publication Language: English 



(30) Priority Data: 

60/229,613 31 August 2000 (31.08.2000) US 

(71) Applicant (for all designated States except US): RYTEC 
CORPORATION LUS/USJ; One Cedar Parkway, P.O. Box 
403, Jackson, WI 53037-0403 (US). 



(72) Inventors; and 

(75) Inventors/Applicants (for US only): GRASSO, Donald, 

= P. [—/US]; 472 Elder Lane, Winnetka, IL 60093 (US). 
= SONN, John [—/US]; 7155 O'Donohue, Hartford, WI 
= 53029 (US). JOHNSON, Steve [—/US]; 32 Saconnet 
= Trail, Little Compton, RI 02837 (US). 



(81) Designated States (national): AE, AG, AL, AM, AT, AU, 

AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, FT, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, 
SL, TJ, TM, TR, TT, TZ, UA, UG, US, UZ, VN, YU, ZA, 
ZW. 



(84) Designated States (regional): ARIPO patent (Gil, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FT, FR, GB, GR, IE, 
IT, LU, MC, NL, PT, SE, TR), OAPI patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GQ, GW, MI,, MR, NE, SN, TD, 
TG). 



Published: 

ii Uhoul international search report and to be republished 
upon receipt of that report 



[Continued on next page] 



\ (54) Title: SENSOR AND IMAGING SYSTEM 



Overview 



00 

On 




(57) Abstract: A system is disclosed having a camera, a processor, and a user interlace. The camera transmits image data responsive 
J to a scene within a field of view. In response to the image data, the processor indicates whether a condition has been satisfied. The 
► user interface is operably connected to the processor and allows a user to select criteria for detection of objects, for indicating criteria 

selected, and for providing visual confirmation that an object has been detected. 



WO 02/19698 A2 1 1 ■ 1 1 1 ■ 1 1 III i lllll IIIIIIIHI HIHIIHII llllllll IHI 



For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations " appearing at the begin- 
ning of each regular issue of the PCT Gazelle. 



WO 02/19698 



PCT/US01/27351 



SENSOR AND IMAGING SYSTEM 

DESCRIPTION 

Related Applications 

This application claims the benefit of U.S. Provisional Patent Application 
Serial No. 60/229,613, filed August 31, 2000, and incorporated herein by reference. 
Technical Field 

5 The present invention relates to sensor and imaging systems, and more 

particularly to a system for providing and interpreting image data. 
Background of the Invention 

Sensor and imaging systems are increasingly in demand in today' s technology 
driven economy. These systems include a camera for viewing objects included within 
10 a field of view. The camera generates image data that is analyzed by a computer to 
determine what, if any, action should be taken in response to the object detected. 

Many recognition systems use two or more cameras for viewing objects 
included within one field of view. In addition to the costs associated with using several 
cameras, these systems require a specific mounting arrangement for each of the cameras. 
1 5 Such systems have a reduced level of reliability over single camera systems because both 
cameras are needed for proper operation. 

Single camera systems are typically mounted at a fixed location and look for 
objects that satisfy, or fail to satisfy, predetenuined criteria. For instance - systems that 
check for structural defects. These systems are incapable of making decisions that are 
20 not already specified. 

Accordingly, a need exists for a sensor and imaging system that, by using an 
image provided by a camera, can decide whether a condition has, or has not, been 
satisfied. 
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Summary of the Invention 

In an embodiment in accordance with the present invention, a system is 
provided having a camera, a processor, and a user interface. The camera transmits image 
data responsive to a scene within a field of view. In response to the image data, the 
5 processor indicates whether a condition has been satisfied. The user interface is operably 

connected to the processor and allows a user to select criteria for detection of objects, 
for indicating criteria selected, and for providing visual confirmation that an object has 
been detected. 

In another embodiment, a control interface is also provided for effecting other 
10 devices. Further, the system provides signals to influence other devices. 

In yet another embodiment, the system provides a signal to open a door upon 
a determination by the processor that a condition has been satisfied. The door is then 
open by a conventional electro mechanical door opener system having a drive motor 
operably connected to the door. 
15 Other features and advantages of the invention will be apparent from the 

following specification taken in conjunction with the following drawings. 
Brief Description of the Drawings 

FIGURE 1 is a block diagram of an embodiment of a sensor and imaging 
system in accordance with the present invention; 
20 FIGURE 2 is a simplified schematic of another embodiment of a sensor and 

imaging system in accordance with the present invention and having a pair of camera 
assemblies connected to a pair of sensor/imaging circuits, respectively; 

FIGURE 3 is an elevation view of a closed door having the camera assemblies 
of FIGURE 2 mounted in proximity thereto; 
25 FIGURE 4 is similar to FIGURE 3 except the door is open; 

FIGURE 5 is a perspective partial assembly view of one of the camera 
assemblies of FIGURE 2 for attachment to a header above the door of FIGURE 3; 

FIGURE 6 is a cross-sectional view of the camera assembly of FIGURE 5; 
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FIGURE 7 is a plan view of a heater assembly attached to the window of the 
camera assembly housing of FIGURE 5; 

FIGURE 8 is a cross-sectional view of the heater assembly, and window, 
taken along plane A-A of FIGURE 7; 
5 FIGURE 9 is a simplified schematic of the camera within the camera 

assembly of FIGURE 8 connected to one of the video processors of FIGURE 2, 

FIGURE 10 is a top, perspective view, of one side of the door of FIGURE 3 
and 4 wherein one of the camera assemblies of FIGURE 2 has a field of view that 
includes the area in front of the door; 
1 0 FIGURE 1 1 is a graphical illustration of the pixel density of the field of view 

within FIGURE 10 as objects are placed further away from the camera assembly; 

FIGURE 12 provides a plurality of safety zone configurations that can, in an 
embodiment, be selected by a user using the membrane keypad of FIGURE 2; 

FIGURE 13 is a plan view of an embodiment of the membrane keypad 
15 depicted in FIGURE 2; 

FIGURE 1 4 is a plan view of a zone positioning method using the membrane 
keypad of FIGURE 13; 

FIGURE 1 5 is a simplified block diagram of one of the object sensor/imaging 
circuits of FIGURE 2 having a Field Programmable Gate Array (FPGA) and a Central 
20 Processing Unit (CPU); 

FIGURE 16 is a top-level simplified block diagram of a processing system 
in accordance with the present invention, having an image processing module within the 
FPGA of FIGURE 1 5, a feature processing module executed by the CPU of FIGURE 1 5, 
and a detection processing module executed by the CPU of FIGURE 15; 
25 FIGURE 1 7 is a simplified block diagram of an initialization sequence for the 

processing system of FIGURE 16, for initializing input parameters and calculating 
related, derived parameters, for initializing detection threshold tables, and for initializing 
detection and feature calculation zones; 
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FIGURE 1 8 is a simplified block diagram of the initialization sequence for 
the processing system of FIGURE 16, for initializing the FPGA image processing 
thresholds and video digitizer settings from user input data via the FPGA serial 
input/output board, and for initializing the feature and detection processing systems; 
5 FIGURE 19 is a simplified block diagram of a video loop processing 

sequence for the processing system of FIGURE 16, utilizing a ping-pong buffer to point 
to, load and unpack reference images into user data; 

FIGURE 20 is a simplified block diagram of the video loop processing 
sequence for the processing system of FIGURE 16, utilizing user data in the CPU to 
10 generate features and detection decisions on a current image frame; 

FIGURE 21 is a simplified block diagram of the video loop processing 
sequence for the processing system of FIGURE 1 6, having diagnostic messages outputted 
at the end of each frame processing and at any point in the video processing; 

FIGURE 22 is a simplified block diagram of a ping-pong system in 
1 5 accordance with the present invention; 

FIGURE 23 is a simplified block diagram of the initialization of the ping- 
pong system depicted in FIGURE 22; 

FIGURE 24 is a simplified block diagram of the ping-pong process loop for 
the ping-pong system of FIGURE 22; 
20 FIGURES 25 is a simplified block diagram of an automatic contrast 

compensation initialization for the processing system of FIGURE 16, for changing the 
video gain in response to image characteristic criteria and time dynamic criteria; 

FIGURE 26 is a simplified block diagram of the automatic contrast 
compensation in a video loop for the processing system of FIGURE 1 6, for changing the 
25 video gain in response to image characteristic criteria and time dynamic criteria; 

FIGURE 27 is a simplified block diagram of the zone initialization sequence 
for the system initialization of FIGURE 1 7, for building zones in real world coordinates, 
generating zone submasks and constructing zone masks; 
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FIGURE 28 is a simplified block diagram of the threshold tables initialization 
sequence for the system initialization of FIGURE 1 7, having an initialize camera intrinsic 
parameters module, a resolution model module and an object model module; 

FIGURE 29 is a simplified block diagram of the image processing module for 
5 the processing system of FIGURE 1 6, having an edge detector; 

FIGURE 3 0 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having a motion detector that uses regions; 

FIGURE 3 1 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having region analysis for shadow and lightbeam 
10 processing; 

FIGURE 32 is a simplified block diagram of the image processing module for 
the processing system of FIGURE 16, having a motion detector that uses edges; 

FIGURE 33 is a simplified block diagram of the feature processing module 
for the processing system of FIGURE 16, for calculating presence, motion, frame and 
15 region features; 

FIGURE 34 is a simplified block diagram of the feature generation system of 
FIGURE 33, having a label module, a calculate global presence features module and a 
calculate zone presence features module; 

FIGURE 35 is a simplified block diagram of the calculate presence feature 
20 system of FIGURE 34, for calculating global presence features; 

FIGURE 36 is a simplified block diagram of the calculate presence feature 
system of FIGURE 34, for calculating zone presence features; 

FIGURE 37 is a simplified block diagram of the feature generation system of 
FIGURE 33, having a label module, a calculate global shadow and lightbeam features 
25 module, and a calculate shadow and lightbeam zone features module; 

FIGURE 38 is a simplified block diagram of the calculate shadow and 
lightbeam region features system of FIGURE 37, for calculating global shadow and 
lightbeam features; 
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FIGURE 39 is a simplified block diagram of the calculate shadow and 
lightbeam region features system of FIGURE 37, for calculating shadow and lightbeam 
zone features; 

FIGURE 40 is a simplified block diagram of a split histogram grey level 
5 analysis for the processing system of FIGURE 1 6; 

FIGURE 41 is a simplified block diagram of the feature generation system of 
FIGURE 33, for calculating frame features; 

FIGURE 42 is a simplified block diagram of the information processing 
module of FIGURE 16, for detecting the presence of an object from features generated; 
10 FIGURE 43 is a simplified block diagram of the information processing 

module of FIGURE 16, for evaluating and updating reference images; 

FIGURE 44 is a simplified block diagram of the information processing 
module of FIGURE 16, for changing threshold values relative to changing background 
values from the field of view; 
15 FIGURE 45 is a simplified block diagram of the information processing 

module of FIGURE 16, for determining the geometric association of edge and zone data 
in a detected object; 

FIGURE 46 is a continuation of FIGURE 45 for the simplified block diagram 
of the information processing module of FIGURE 16, and includes testing region edges 
20 to determine zone intersections in detected objects; 

FIGURE 47 is a continuation of FIGURE 46 for the simplified block diagram 
of the information processing module of FIGURE 16, and includes evaluating region 
scores of zones to determine zone intersections in detected objects; 

FIGURES 48 and 49 are tables for defining various image operations; 
25 FIGURE 50 is a representative example of a background or reference image; 

FIGURE 5 1 is similar to FIGURE 50 except it is a current image wherein an 
object has entered the field of view; 
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FIGURE 52 is the difference between subtracting the reference image in 
FIGURE 50 from the current image in FIGURE 51 ; 

FIGURE 53 is the difference between subtracting the current image in 
FIGURE 51 from the reference image in FIGURE 50; 
5 FIGURE 54 is a resulting image after thresholding and shape filtering the 

image in FIGURE 52; 

FIGURE 55 is a resulting image after thresholding and shape filtering the 
image in FIGURE 53; 

FIGURE 56 is a resulting image after completing a logical OR operation on 
10 the images in FIGURE 54 and FIGURE 55; and 

FIGURE 57 is similar to FIGURE 56 except that regions within the image of 
FIGURE 56 are labeled for classification. 
Detailed Description of the Preferred Embodiment 

While this invention is susceptible of embodiments in many different forms, 
15 there is shown in the drawings and will herein be described in detail preferred 
embodiments of the invention with the understanding that the present disclosure is to be 
considered as an exemplification of the principles of the invention and is not intended 
to limit the broad aspect of the invention to the embodiments illustrated. 

Turning to FIGURE \, a block diagram of an embodiment of a system in 
20 accordance with the present invention is depicted. The system 10 includes aspects 

directed to image analysis 12, presence detection 14, sensor technology 16, electronics 
18, device control 20, camera pose 22, camera technology 24, image analysis 26, and 
market applications 28. 

Turning to FIGURE 2, a simplified schematic of another embodiment of a 
25 system in accordance with the present invention is depicted. The system 1 10 includes 

a pair of camera assemblies 112,113 operably connected, respectively, to a pair of 
sensor/imaging circuits 114,115. The sensor/imaging circuits 114,115 are operably 
connected to an I/O display board 116 that is connected to a keypad 118. In a further 
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embodiment, the system 110 can include an electro mechanical door opener system 117 
having a drive motor 119 operably connected to a door 121 (FIGURES 3 and 4). 

The cameras assemblies 112 andll3 can include charge coupled devices 
(CCD), or the like, having preferably a wide-angle lens, and capable of transmitting 
5 image data to the sensor/imaging circuits 1 14 and 115, respectively. The image data 

corresponds, respectively, to a scene within each camera's field of view. 

The sensor/imaging circuits 1 1 4, 1 1 5 process the image data for determining 
whether a user selected condition has been satisfied. The user selected conditions are 
selected via a man-machine interface comprising the I/O display board 116 and the 

10 membrane keypad 118. In an embodiment, the man-machine interface is operably 

connected to the sensor/imaging circuits 1 14,1 15 and allow a user to select criteria for 
detection of obj ects, for indicating criteria selected, and for providing visual confirmation 
that an object has been detected. 

FIGURE 3 provides a cross sectional elevation view of a doorway 120 that 

1 5 is closed by a door 121. Mounted in proximity to the doorway 1 20 and on opposite sides 
thereof are the camera assemblies 112,113 of FIGURE 2. According, the camera 
assemblies 112 and 1 13 have afield-of-view 122 and 123, respectively, on opposite sides 
of the doorway. Preferably, the field-of-view of at least one camera assembly includes 
the doorway 120 when the door 121 is open as shown in FIGURE 4. Further, the field- 

20 of-views 122 and 123 overlap about the doorway 120. 

Turning to FIGURE 5 a perspective partial assembly view is depicted of one 
of the camera assemblies 1 12,1 13 of FIGURE 2 for attachment to a header above the 
door 121 of FIGURE 3. The camera assembly includes a housing assembly 131 having 
a window 132. FIGURE 6 provides a cross-sectional view of the camera assembly of 

25 FIGURE 5. Mounted within the housing assembly 1 3 1 is a camera 1 34 having a field-of- 

view 136 and a heater assembly 140. In a further embodiment, a power supply (not 
shown) can be mounted within the heater assembly for supplying power to the camera 
134. 



WO 02/19698 



PCT/US01/27351 



FIGURES 7 and. 8 provide a plan view and a cross-sectional view, 
respectively, of the heater assembly 140. In an embodiment, the heater assembly 140 
adjoins the window 132 of the camera assembly housing 131, preferably made of cast 
metal or the like, and includes a ceramic resistor 142 sandwiched between a pair of 
5 conductive terminals 144,145. The ceramic resistor 142 is generally ring shaped and 

constructed of a material that exhibits what is commonly referred to as a PTCR (positive- 
temperature coefficient or resistance) or PTC effect. Likewise, the conductive terminals 
144,145 are generally ring shaped and constructed of an electrically and thermally 
conductive material such as copper. 
10 In an embodiment, annular apertures 146 extend through the axis of the 

ceramic resistor 142 and the conductive terminals 144,145. The apertures 146 have 
substantially identical outer circumferences and are concentrically aligned with each 
other. 

Preferably, the outer perimeter 148 of conductive terminal 145 includes a 
15 plurality of ears 1 50 extending outwardly therefrom. Extending through each ear 1 50 is 

an aperture 152 for extending an attachment screw 154 (FIGURE 9) therethrough. 

Turning to FIGURE 9, the heater assembly 1 40 is mounted within the housing 
1 3 1 of the camera assembly. As indicated previously, attachment screws 1 54 couple the 
conductive terminal 145 of the heater assembly 140 to a mounting surface, or 
20 alternatively, mounting posts 156 that inwardly extend into the housing 131 and are 

integrally attached thereto. 

The camera assembly housing 131 includes an aperture 158 that allows the 
camera's field-of-view 136 to extend outside of the housing. The window 132 is 
mounted over the aperture 1 58 to prevent contaminants such as dirt and moisture from 
25 entering the camera assembly. 

Preferably, the window 1 32 is sandwiched between the thermally conductive 
terminal ring 145 of the heater assembly 140 and an annular gasket 160 made of a 
resilient material and adjoining against the inner surface 162 of the camera assembly 
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housing 131 about aperture 158. In an embodiment, the window 132 is made of a 
visually transparent material such as borosilicate glass 

The camera 134 generates image data or electronic pixel data 218 
representative of what is observed in the camera's field-of-view 136. In an embodiment, 
5 the image data 218 is analyzed by one of the video processors 1 14,1 15 (FIGURE 2) for 

determining what, if any, action should be taken in response to what is detected in the 
camera's field-of-view 136. 

Within the camera assembly housing 1 3 1 , the terminals 1 44, 1 45 of the heater 
assembly 140 are coupled to a voltage source 166 for maintaining a voltage potential 

1 0 across the ceramic resistor 1 42 . The heat generated by the ceramic resistor 1 42 as current 

flows therethrough is dissipated through conductive terminal 145 arid onto the window 
132. In an embodiment, enough heat is provided to maintain the window 132 at a 
temperature above the dew-point of the air outside of the camera assembly housing 131. 
For instance, the heater can activate at about 87 °F and deactivate at about 106°F. 

15 As will be appreciated by those having skill in the art, the use of a ceramic 

resistor 142 eliminates the need for a mechanical thermostat or the like since the resistor 
material exhibits a positive-temperature coefficient. The elimination of a thermostat 
increases the reliability of the heater and reduces the amount of noise placed on voltage 
supplies as a result of switching. 

20 Turning back to FIGURE 2, in an embodiment the camera assemblies 112,113 

can include an illumination source (not shown), such as a lightbulb, light emitting diodes 
within the humanly visible or non- visible spectrum, or the like, to illuminate the field of 
view. The output of the camera assemblies 112,113 can be received by a removably 
coupled display (not shown) for allowing a user to view check that each camera 

25 assemblies' field of view is properly configured. 

Turning to FIGURE 10 a top perspective view is provide of one side of the 
door 1 2 1 of FIGURE 3 and 4 wherein one of the camera assemblies (in particular camera 
assembly 112) of FIGURE 2 has a field of view 122 that includes the area in front of the 
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door. The camera assembly 122 provides image data to the video processor 114 
(FIGURE 2) which, in an embodiment, superimposes a safety zone 168 and, if desired, 
one or more activation zones 170. Preferably, when an object is detected by either of the 
video processors 1 1 4, 1 1 5 to be in a safety zone 1 68, the door 1 2 1 is opened and remains 
5 so until the object is outside of the safety zone 168. Further, when an object is detected 

by either of the video processor 1 14,1 15 to enter an activation zone 168, the door 121 is 
opened are remains open for a period of time after the object stops moving in the 
activation zone or leaves the activation zone. 

Preferably, the safety zone 168 is maintained in an area immediately 

1 0 suiTounding the door 1 2 1 to prevent the door from closing when a person or obj ect is in 

the immediate vicinity of the door. Moreover, the activation zone(s) 168 open the door 
when a person or vehicle approaches the door 121. A failsafe system can also be 
provided to open the door 121 whenever there is a loss of illumination within the field 
of view, severe illumination changes, electronics failure, camera knocked ajar, or the 

1 5 camera lens is obscured. 

FIGURE 1 1 depicts the pixel density of the field of view 122 as objects are 
placed further away from the camera assembly 122. FIGURE 12 provides a plurality of 
safety zone configurations that can, in an embodiment, be selected by a user using the 
membrane keypad 1 18 (FIGURE 2). 

20 FIGURE 13 provides an plan view of an embodiment of the membrane 

keypad 1 1 8 depicted in FIGURE 2. The membrane keypad 1 1 8 and the I/O display board 
116 (FIGURE 2) provide a user interface or man-machine interface for a user to setup 
the system (FIGURE 2). In an embodiment, the user interface allows a user to define the 
door type, and dimensions, and establish the size, location, and type of detection zones 

25 the system 110 will use. Preferably, the user interfaces allows a user to enter data to 

define at least one control zone parameter (i.e., activation zone or safety zone) from the 
group consisting of area, location, shape, number of control zones, and control criteria. 
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As indicated previously, the system 110, via the user interface, has the 
capability to define at least a portion of an image received by the camera assemblies as 
a control zone. In an embodiment, the system 110 has the capability to choose 
coordinates from all pixel coordinated by direct access within the control zone. 
5 Alternatively, the system 110 has the capability to choose from multiple predefined 

zones. Moreover, the system 1 10 can have the capability to put real objects in the field 
of view so as to delineate boundary coordinates and the real objects become part of the 
image data. 

In an embodiment, the user interface has three modes of operation: parameter 

l o edit mode, run mode, and diagnostic mode. In parameter edit mode, a user can input or 
modify configuration parameters, using touch keypad buttons 610, 622, 624, and 626, 
such as the door model, English or metric units, camera heights and distance from the 
door. In the run mode, the system 1 10 is activated. As such, the system 1 10 processes 
images from the cameras 112,113 and outputs safety and activation zone indication 

15 signals through the I/O board 116, and displays status information on the display LEDs 
6 1 4, and 616. In the diagnostic mode, additional information regarding the status of the 
system 1 10 is made available via an I/O port (not shown). 

FIGURE 14 provides a diagram depicting movement of zones using the 
membrane keypad of FIGURE 13. In an embodiment, a user can move activation zones 

20 to various locations within the field-of-view of the camera assemblies 1 12, 1 1 3 . 

FIGURE 1 5 provides a simplified block diagram of one of the sensor/imaging 
circuits 1 14,1 1 5 of FIGURE 2. In an embodiment, the sensor/imagining circuits 1 14, 1 1 5 
are substantially similar in physical design and include a Field Programmable Gate Array 
(FPGA) 124, a Central Processing Unit (CPU) 125, and a video digitizer 126. 

25 In an embodiment, the video digitizer 126 receives an analog image signal 

from one of the cameras, digitizes the analog image signal, and transmits the digitized 
image signal to the field programmable gate array 124. 
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As explained in detail further herein, the field programmable gate array 124 
is programmed to perform one or more image processing operations in response to the 
digitized image signal received. In an embodiment, these operations include comparing 
predetermined traits of the digitized image signal with one or more previously received 

5 digitized image signals to provide composite image data. In response to the composite 

image data, the central processing unit 125 determines whether one or more conditions 
have been satisfied. 

Operably coupled to the video digitizer 126 is a temperature sensor 128 
having an output correlating to the temperature of the video digitizer. Upon an indication 

1 0 by the temperature sensor of a condition whereby the video digitizer 1 26 is not operating 

within a predetermined temperature range or limit, reset command is issued by a reset 
circuit 130 whereby the system 1 10 maintained in a reset state until the temperature of 
the video digitizer returns to within the predetermined temperature range or limit. 

In an embodiment, the FPGA 124 performs a relatively high-rate pixel 

15 processing in order to unburden the CPU 125 and achieve a desired video processing 
frame rate. This hardware architecture balance reduces overall system cost by removing 
the cost associated with an adequately fast CPU chip. A further frame rate speed up can 
be achieved by using the FPGA and CPU processing simultaneously in parallel. This 
parallel processing is accomplished by FPGA pixel processing the next frame during the 

20 interval that the CPU is data processing the current frame. Thus, the new FPGA output 

is immediately available to the CPU process when the CPU finishes the current frame 
data processing. This process structure requires the ability to maintain two independent 
sets of data, and is referred to later herein as ping/pong control. 

Turning to FIGURE 16, a top-level simplified block diagram is depicted of 

25 a processing system in accordance with the present system. The system preferably 
includes an image processing module 2 1 2 within the FPGA 1 24 of FIGURE 1 5, a feature 
processing module 214 executed by the CPU 125 of FIGURE 15, and a detection 
processing module 216 also executed by the CPU 125 of FIGURE 15. 
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In an embodiment, the image processing module 212 receives stored 
electronic pixel data 2 1 8 comprising current image data 220, reference image data 222, 
reference edges data 224, and previous image data 226. Preferably, the current image 
data 220 is the most recently taken image (i.e., taken at t), the previous image data 226 
5 is the next most recently taken image data (i.e., taken at t+1), and the reference image 

data 222 is the oldest of the taken image data (i.e., taken at t+l+x). Moreover, as 
explained in detail further herein, the reference edges data 224 consists of edge data 
extracted from the reference image data 222. 

The image processing module 212 also receives parameter data 228 from the 
10 man-machine interface (i.e., membrane keypad 118 and I/O display board 116 of 

FIGURE 2). As explained in detail further herein, the parameter data 228 includes 
information pertaining what areas (i.e., control zones) that a detected object within the 
field of views (1 12 and 123 of FIGURES 2 and 3) is to result in opening of the door 121 
(FIGURES 2 and 3). 

15 As explained in detail further herein, in response to the electronic pixel data 

218 and the parameter data 228, the image processing module 212 produces derived 
image data 230 comprising edge segmentation, motion segmentation, and region 
segmentation. 

The feature processing module 214 receives the derived image data 230 and 
20 the parameter data 228. As explained in detail further herein, the feature processing 

module 2 1 4 produces, in response to the image data 23 0 and parameter data 228, feature 
data 232 comprising edge features, motion features, region features, and frame features. 

The detection processing module 216 receives the feature data 232 and the 
parameter data 228. In response to the data, the detection processing module 216 
25 produces control signals 234 comprising a detection signal for opening and closing the 

door 121 (FIGURES 2 and 3), reference updates, and adaptive thresholds. 

Turning to FIGURES 17 and 1 8, a simplified block diagram is depicted of an 
initialization sequence for the processing system of FIGURE 16. The initialization 
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sequence 234 includes an initialize parameters step 23 6, an initialize threshold tables step 
238, an initialize zones step 240, an initialize FPGA step 242, an initialize video digitizer 
step 244, and an initialize video system step 246. 

The initialize parameters step 236 includes initialization of the man-machine 
5 interface and constant data and derived parameters. During initialization of the man- 

machine interface, user entered data is read and stored into memory. Constant data is 
also loaded into memory along with derived parameters relating to control zones for 
opening and closing the door. 

Thus, as indicated above, upon application of power to the system, the 

1 0 initialize parameter module 236 initiates the initialization of the man-machine interface 

(i.e., membrane keypad 118 and I/O display board 116 of FIGURE 2), constant data, and 
derived parameters. The initialize threshold table module 238 initiates the initialization 
of the area threshold, maps from the camera geometry and resolution models. These 
maps are used to determine minimum and maximum pixel characteristics of obj ects such 

15 as people and vehicles such as, for example, forklifts. 

The initialize zones module 234 initiates the initialization of the control zones 
whereupon data associated with user or predefined safety zones and activation zones is 
complied. The initialize FPGA 242 and the initialize video digitizer 244 initiates the 
initialization of the FPGA 124 (FIGURE 15) and the video digitizer 126 (FIGURE 15), 

20 respectively. In particular, the control status register (CSR) and image buffers pointer 

are initialized during FPGA initialization. Further, the video digitizer is initialized by 
constructing the required instructions and sending them, via the FPGA serial I/O. 

The initialize video system 246 initiates the initialization of the CPU 125 
(FIGURE 1 5). In particular, the first ping-pong data set is selected. Next, the FPGA is 

25 instructed to capture a video frame. Four images (reference images) are then initialized - 

a grey level reference (R), an edge reference (GER) and an archive edges reference. The 
FPGA is then instructed to process these data. The FPGA outputs are retrieved from the 
FPGA into the Bank 0 database. 
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After the system is initialized, the system operates in a video processing loop 
depicted in the simplified block diagrams of FIGURES 19-22. In an embodiment, the 
video processing loop 250 includes a get next frame step 252, a retrieve FPGA data step 
254, a calculate all features step 214, a calculate detection & control step 258, a save 
5 FPGA data step 260, and a write diagnostics step 262. 

Within the video processing loop 250, the CPU 125 process use the current 
ping/pong buffer to point to, load and unpack that data into a third database - the user 
data - bank U. This data is used in the CPU process later to generate features and 
detection decisions on the current frame. Preferably, at the same time, the CPU process 

10 starts the FPGA capture and process activity on the FPGA 124. While the CPU is 

processing features for the current frame, the FPGA is computing image data for the next 
frame. The detection and control activity sends the safety and activate signals out 
through the FPGA serial I/O interface. The CPU feature and detection processing takes 
longer than the FPGA computations. When the CPU finishes the current frame, the 

15 FPGA data is retrieved to the opposite bank (e.g., Bank 1 if processing Bank 0). 

Diagnostic messages can be output at the end of each frame processing, as well as at any 
point in the video processing. The process then loops to set Bank U to the new current 
bank (Bank 0 or Bank 1), and the FPGA is again initiated. 

Turning to FIGURE 22, a simplified block diagram is provided of an 

20 embodiment of a system resources allocation method, or ping/pong control, in 

accordance with the present invention. As stated previously, the video processing system 
110 includes an FPGA 124 for image processing. This results in the production of 
derived image data 230 comprising edge segmentation, motion segmentation, and region 
segmentation. 

25 Once the derived image data 230 is produced, it is preferably stored within 

one of a plurality of memory banks 230a,230b and then provided, via switching, for 
feature processing. Accordingly, the derived image data 230 provided to the feature 
processing module 214 is static. However, the FPGA 124 continuously processes the 
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electronic pixel data 218 and loads the results of the image processing, via switching, 
into the memory bank not currently accessible to the processing module 214. 
Accordingly, the derived image data 230 within the memory banks is accessible to the 
feature processing module via switched between the memory banks 23 0a,23 Ob on a first- 
5 in-first-out basis. 

Preferably, two memory banks 230a and 23 Ob are provided. Turning back to 
FIGURE 19, the get next frame step 252 provides for the capture and processing of 
electronic pixel data 2 1 8 by the image processing module 2 1 2 within the FPGA 1 24. In 
particular, a control and status register (CSR) is used for selection of the memory banks 
10 230a,230b and to set capture & process bit. 

The retrieve FPGA data step 254 provides for obtaining the static data within 
the memory banks for processing of the static data during the calculating all features step 
256. In particular, temporary storage registers and counters are reset, and the static data 
is unpacked to provide the derived image data 230 for processing by the feature 
1 5 processing module 2 1 4 (FIGURE 1 6). 

In an embodiment, and as explained in detail further herein, the feature 
processing module 214 (FIGURE 16) performs the calculate all features step 256, in 
response to the derived image data 230 (FIGURE 1 6) and parameter data 228 (FIGURE 
16). In particular, the calculate all features step 256 produces feature data 232 (FIGURE 
20 16) comprising edge or P features, motion features, shadow and lightbeam or region 

features, and frame features. 

Further, the detection processing module 216 performs the calculate detection 
& control step 25 8, in response to the feature data 232 (FIGURE 1 6) and parameter data 
228 (FIGURE 16). In particular, frame analysis fault flags are evaluated, detection logic 
25 is executed, reference frame update requires are evaluated, automatic adaptive thresholds 

are evaluated, and motion reference data is updated. 

The save FPGA data step 260 occurs once the FPGA 124 (FIGURE 15) 
provides an interrupt to the CPU 125 (FIGURE 15) indicating that the FPGA has 
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completed processing of another image frame and the data is ready for processing by the 
feature processing module (FIGURE 16). Moreover, if desired, the write diagnostics step 
262, which is performed by CPU 125 (FIGURE 15), can store within memory one or 
more messages regarding how the processing of the previous frame data progressed. The 
5 video processing loop 250 then preferably continues back to the get next frame step 252. 

Turning to FIGURE 23, a simplified block diagram is provided of an 
embodiment of an initialization procedure for the ping-pong system of FIGURE 22. In 
an embodiment, the ping-pong initialization procedure 264 includes an initialize ping- 
pong addresses module 265, a capture video buffer module 266, a capture frame module 

10 267, an initiate video references module 268, a process video module 269, a process 

frame module 270, and a retrieve FPGA data module 271 . 

FIGURE 24 provides a simplified block diagram of the ping-pong process 
loop 272 for the ping-pong system of FIGURE 22. The top of the loop 272 shows the 
CPU activity while the bottom shows the FPGA activity (not in time scale), with the 

15 associated serial I/O and CSR messaging. 

FIGURES 25 and 26 illustrate the details of the ping/pong activity 
(initialization & video loop) in an alternative embodiment having an automatic contrast 
circuit (ACC). The ACC can be used to improve system detection performance due to 
slowly changing lighting conditions. The ACC does this by changing the video gain in 

20 response to image characteristic criteria and time dynamic criteria. The ACC maximizes 

gain while preventing too much image white saturation. After a gain change, the video 
system is reinitialized. 

The ACC, when enabled by user input, functions during initialization to find 
the best starting gain by iterating and testing the image result. When a gain is found 

25 which satisfies established criteria, iterating stops, and the process continues to the video 

loop with the selected gain. The ACC also functions at the beginning of the video loop, 
but does not iterate to fine a satisfactory gain. Only a single gain change is performed 
in the loop per frame. The gain change and consequent video system initialization take 
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a much shorter time than a frame time (100 ms). The decision to require a gain change 
in the video loop is controlled by criteria calculated in the detection and control portion 
of the CPU activity. The criteria can include aging, zone activity, and long and short 
time-constant filters. 

5 FIGURE 27 provides a simplified block diagram of the zone initialization 

sequence 240 for the system initialization of FIGURE 17. The zone initialization 
sequence 240 results in the building of zones in real world coordinates, generating 
control zone submasks and constructing control zone masks. FIGURE 28 provides a 
simplified block diagram of the threshold tables initialization sequence 238 for the 
10 system initialization of FIGURE 17. The threshold tables initialization sequence 238 

result in the initialization of camera intrinsic parameters, the resolution model, and the 
object model. 

Turning to FIGURE 29 (and referencing FIGURES 48 and 49), a simplified 
block diagram is provided of the image processing module for the processing system of 

15 FIGURE 16 which includes an edge detector 301. The edge detector 301 preferably 

includes a modified Sobel operator module 302, a positive difference module 304, a 
threshold module 306, an erode module 308, and a label I module 310. 

In an embodiment, the modified Sobel operator module 302 receives current 
(B) image input 3 12 and generates the edge image (GEB) 314 from the current input 

20 image. A reference image (GER) 316, initialized in the CPU, is subtracted from the 

current edge image in the positive difference operator module 3 04, where negative values 
are set to zero. The grey-level edge image is thresholded 306, eroded 308, and labeled 
310. The output of the label I operator 3 10 is a 16-bit labeled image 3 1 8, an equivalence 
table 320, and counts of the number of labels used 322 and the number of entries in the 

25 equivalence table. Counts of the number of set pixels in the binary input 324 and output 

326 of the erode operator 308 are also output to the CPU, completing the edge image 
processing. 
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Label I operator 3 1 0 is used in each thread of the image processing. Label I 
3 10 is the first part of a two step process used to produce the labeling of the connected 
components of the binary input. Label 1310 passes a 2x2 kernel over the binary input 
image beginning with the upper left of the image. The elements of the kernel are 
5 identified as follows: 

B C 
A X 

If the binary pixel in X is zero, the output is zero. If X is set, the labels B, A, C are 
scanned in that order. If all of B, A, C are non-zero, the next value of a label counter is 
10 output at X and the counter is incremented. If any B, A, C are non-zero, the label 
operator is the value of the first non-zero label. If more than one of B, A, C is non-zero, 
the first non-zero value is output. If any of the remaining non-zero labels is different 
from the output value, the output value and the different value are written to an 
equivalence table. 

1 5 Turning to FIGURE 3 0 (and referencing FIGURES 48 and 49), a simplified 

block diagram of the image processing module for the processing system of FIGURE 1 6, 
having a motion detector that uses regions 340. The system 340 preferably includes 
positive difference modules 342 and 344, threshold modules 346 and 348, and dilate 
modules 350 and 3 52 for both the current (B) input image (n) and previous (A) image (n- 

20 1 ) respectively. The output is passed to the inclusive OR module 354, erode module 356, 

and Label I module 358. 

The positive difference of the current grey-level input image (B) 360 and the 
previous image (A) 362 is thresholded 346 and 348 and dilated 350 and 352, as well as 
the positive difference of A and B 342 and 344. The results are inclusively ORed 354. 

25 The resulting binary image is labeled as in the edge case (FIGURE 29), and the results 
are passed to the CPU. 

The grey-level edge image is thresholded, eroded 356, and labeled 358. The 
output of the label I operator 358 is a 16-bit labeled image 364, an equivalence table 366, 
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and counts of the number of labels used 368 and the number of entries in the equivalence 
table. Counts of the number of set pixels in the binary input 370 and output 372 of the 
erode operator 356 are also output to the CPU, completing the motion detector image 
processing using regions. 
5 Label I operator 358 is used in each thread of the image processing. Label 

I 358 is the first part of a two step process used to produce the labeling of the 
connected components of the binary input. Label I 358 passes a 2x2 kernel over the 
binary input image beghming with the upper left of the image. The elements of the 
kernel are identified as follows: 
10 B C 

A X 

If the binary pixel in X is zero, the output is zero. If X is set, the labels B, A, C are 
scanned in that order. If all of B, A, C are non-zero, the next value of a label counter is 
output at X and the counter is incremented. If any B, A, C are non-zero, the label 
15 operator is the value of the first non-zero label. If more than one of B, A, C is non-zero, 

the first non-zero value is output. If any of the remaining non-zero labels is different 
from the output value, the output value and the different value are written to an 
equivalence table. 

Turning to FIGURE 31 (and referencing FIGURE 48 and 49), a simplified 
20 block diagram of the image processing module for the processing system of FIGURE 1 6, 

having region analysis for shadow and lightbeam processing. The system 380 preferably 
includes positive difference modules 382 and 384, threshold modules 386 and 388, and 
dilate modules 390 and 392 for both the current (B) input image 394 and reference (R) 
input 396, respectively. The output is passed to the inclusive OR module 396, erode 
25 module 400, and Label I module 402. The output of the label I operator 402 is a 1 6-bit 
labeled image 404, an equivalence table 406, and counts of the number of labels used 408 
and the number of entries in the equivalence table. Counts of the number of set pixels 
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in the binary input 410 and output 412 of the erode operator 402 are also output to the 
CPU, completing the motion detector image processing using regions. 

The system image processing region analysis detection operation is analogous 
to the motion detection operation of FIGURE 30 except that instead of using the 
5 immediately previous image input A, it uses a previous image called the reference image 

396 which is updated on CPU command as a copy of the current input image. The region 
analysis thread also produces a difference image 414 and a 16-level histogram 416 for 
CPU use. 

Turning to FIGURE 32 (and referencing FIGURES 48 and 49), a simplified 

1 0 block diagram of the image processing module for the processing system of FIGURE 1 6, 

having a motion detector that uses edges 420. The system 420 illustrates a selectable 
alternative motion detection image processing operation similar to the edge detection 
operation of FIGURE 29, except mat it takes the current edge image (GEB) 422 and the 
previous edge image (GEA) 424 as input to the positive difference module 426. The 

15 positive difference of GEB 422 minus GEA 424 is thresholded 428, eroded 430 and 
labeled 432 as in FIGURES 19, 20, and 21 . 

The output of the label I operator 432 is a 16-bit labeled image 434, an 
equivalence table 436, and counts of the number of labels used 438 and the number of 
entries in the equivalence table. Counts of the number of set pixels in the binary input 

20 440 and output 442 of the erode operator 43 0 are also output to the CPU, completing the 

system image processing having a motion detector that uses edges. 

Turning to FIGURE 33, a simplified block diagram of the feature processing 
module 450 for the processing system of FIGURE 16, for calculating presence, motion, 
frame and region features. The system data processing, calculating all features, is 

25 performed in the FPGA to unburden the CPU and achieve the desired processing rate. 

The presence (P) or edge feature module 452 and the shadow and lightbeam 
(SL) or region feature module 45 8 calculations are quite similar to the point of generating 
the edge/region score discounts. Moreover, within the P feature 452 and SL feature 458 
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calculations, the global calculations are very similar to the zone calculations. The zone 
calculations restrict the spatial range of feature calculations for each zone using the 
associated zone mask. The results of the P and SL feature calculations are stored in a 
database (feature tables) for use in detection determination 460. 
5 Frame features 456 are calculated differently than P and SL features. Frame 

features 456 are not features of objects, but of the input grey level image and the current 
edge image. Frame statistics are computed in order to draw inferences about conditions 
of the camera and video system integrity. Frame statistics are also used to condition 
some detection variables that act as adaptive thresholds. Three fault flags can be set by 

10 the calculate frame features module 456: illumination fault flag, obscure fault flag, and 

ajar fault 462. Each of these faults 462 is determined through associated metrics. The 
illumination fault is controlled by evaluating the modified Kuiper statistic, the uniform 
centered mean, and the variance of the grey level input. The obscure and ajar faults use 
the current and archive edges to detect whether the camera and/or video system have 

1 5 become obscured or knocked ajar. 

The system 450 will not update the reference if any motion is detected in any 
zone. To determine if there is motion in each zone, the labeled edge image counts the 
number of non-zero pixels in the labeled zone, calculated in the presence P feature 
module 452. The non-zero pixels become motion pixels, calculated in the motion M 

20 feature module 454. The system 450 counts the non-zero pixels in the motion labeled 

image to verify if the zone motion pixels in each zone is greater than zero (0). The 
system 450 counts the non-zero pixels in the zone detection mask for accumulation in 
the count. 

Turning to FIGURE 34, a simplified block diagram of the feature generation 
25 system 470 of FIGURE 33, having a label module 472, a calculate global presence 

features module 474 and a calculate zone presence features module 476. 

The label module 472 receives presence input in the form of labeled edge 
image 478, equivalence tables 480, and label and conflict counts 482. The label module 
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472 resolves pixel labeling conflicts within the region, it replaces labels with region 
numbers, it makes an area call, renumbering regions with sequential indices, and re- 
indexes the region again, passing data related to the number of regions and regions image 
to the calculate global presence features model 474. 
5 The calculate global presence features model 474 uses the regions image 484, 

the number of regions 486 and current edges (GEB) 488 to create a global feature table. 
The global feature table is first initialized, regions are labeled as to area, mean grey level 
intensity, histogram, and centroid. The region is then recalculated for variance of grey 
level and centroid, listing the features (global, safety, . . .) of the pixels within the region. 

10 The calculate P feature zones module 476 takes the aforementioned regions 

image 484, number of regions 486, the current edges (GEB) 488 and creates a zone 
feature table using zone mask and rectangle 490. The system 470 determines motion in 
zones by calculating detection in a safety zone, a secondary safety zone, a door zone, a 
first activation zone, a second activation zone, a first guard zone, and a second guard 

15 zone. 

Turning to FIGURE 35, a simplified block diagram of the calculate presence 
(edge) feature system of FIGURE 34, for calculating global presence features module 
500. 

The global P features are calculated first by initializing the edge counts of the 
20 feature table 502. The global extent of the image is calculated in the image first pass 

module 504. Area, centroid, mean, histogram and edge counts are accumulated and put 
through the image second pass module 506 where a second central moments and variance 
is accumulated. The feature table pass module 508 calculates the derived features 
including the spread, elongation, orientation, and ellipse shape of the region. The 
25 calculate region scores module 510 determines door rejection, edge shape suppression, 

and edge grey level suppression. A score comes from the grey level variance of the 
region and a discount is applied to the score. After region scores are calculated 510, the 
next region is looped through the calculations of the feature table pass module 508. 
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Turning to FIGURE 36, a simplified block diagram of the calculate presence 
feature system of FIGURE 34, for calculating zone presence features. 

The zone presence P features are calculated first by initializing the zone 
counts of the feature table 522. The global extent of the zone is calculated in the zone 
5 first pass module 524. The zone is calculated to determine if pixels are in the selected 

zone mask. Area, centroid, mean, and histogram are also accumulated and put through 
the image second pass module 526 where a second central moments and variance is 
accumulated. The feature table pass module 528 calculates the derived features including 
the spread, elongation, orientation, and ellipse shape of the region. The calculate region 
10 scores module 530 determines door rejection, area proportion suppression, edge shape 

suppression, and edge grey level suppression. After region scores are calculated 530, the 
next region is looped through the calculations of the feature table pass module 528. 

Turning to FIGURE 37, a simplified block diagram of the feature generation 
system 540 of FIGURE 33, having a label module, a calculate global shadow and 
15 lightbeam (SL) features module, and a calculate shadow and lightbeam zone features 

module. 

The label module 542 receives presence input in the form of labeled edge 
image 544, equivalence tables 546, and label and conflict counts 548. The label module 
542 resolves pixel labeling conflicts -within the region, it replaces labels with region 

20 numbers, it makes an area call, renumbering regions with sequential indices, and re- 

indexes the region again, passing data related to the number of regions and regions image 
to the calculate global presence features model 550. 

The calculate global presence features model 550 uses the regions image 552, 
the number of regions 554 and the current difference image to create a global feature 

25 table. The global feature table is first initialized, regions are labeled as to area, mean 
grey level intensity, histogram, and centroid. The region image is then recalculated for 
variance of grey level and centroid second movements, listing the shape features of the 
image within the region. 
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The calculate SL feature zones module 558 takes the aforementioned regions 
image 552, number of regions 554, the current edges (GEB) 560 and creates a zone 
feature table using zone mask and rectangle 562. The system 540 determines motion in 
zones by calculating detection in a safety zone, a secondary safety zone, a door zone, a 
5 first activation zone, a second activation zone, a first guard zone, and a second guard 

zone. 

Turning to FIGURE 38, a simplified block diagram of the calculate shadow 
and lightbeam region features 570 system of FIGURE 37, for calculating global shadow 
and lightbeam (SL) features. 

10 The global SL features are calculated first by initializing the edge counts of 

the feature table 572. The global extent of the image is calculated in the image first pass 
module 574. Area, centroid, mean, histogram and edge counts are accumulated and put 
through the image second pass module 576 where a second central moments and variance 
is accumulated. The feature table pass module 578 calculates the derived features 

1 5 including the spread, elongation, orientation, ellipse shape factor of the region, modified 

kniper statistic and mapped mean and variance. The calculate region scores module 580 
determines the SL score with region suppression from shadow and light beam discount, 
shape discount, and area discount and with transient suppression. After region scores are 
calculated 5 80, the next region is looped through the calculations of the feature table pass 

20 module 578. 

Turning to FIGURE 39, a simplified block diagram of the calculate shadow 
and lightbeam region features 590 system of FIGURE 37, for calculating shadow and 
lightbeam (SL) zone features. 

The zone SL features are calculated first by initializing the zone counts of the 

25 feature table 592. The global extent of the zone is calculated in the zone first pass 
module 594. The zone is calculated to determine if pixels or zone rectangle are in the 
selected zone mask. Area, centroid, mean, and histogram are also accumulated and put 
through the image second pass module 596 where a second central moments and variance 
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is accumulated. The feature table pass module 598 calculates the derived features 

including the spread, elongation, orientation, ellipse shape factor, modified kniper 

statistic, and mapped means and variance of the region. The calculate region scores 

module 600 determines the SL score with region suppression from shadow and light 
5 beam discount, shape discount, and area discount and with transient suppression. After 

region scores are calculated 600, the next region is looped through the calculations of the 

feature table pass module 598. 

FIGURE 40 provides a simplified block diagram of a split histogram grey 

level analysis 700 for the processing system of FIGURE 1 6, and in particular the feature 
10 processing module 214. In an embodiment, the split histogram grey level analysis 700 

can be provided to assists in the determination of region features. 

FIGURE 41 provides a simplified block diagram of the feature generation 

system 750 of FIGURE 33, for calculating frame features. The system 750 include a 

frame analysis initialization module 752, a compute frame statistics module 754, an 
15 interpret frame illumination metrics module 756, an interpret obscure metrics module 

758, an interpret ajar metrics module 760, and an update FA filter 762. 

FIGURE 42 is a simplified block diagram of the information processing 

module of FIGURE 16, for detecting the presence of an object from features generated. 

The detection sequence includes an evaluate fault flags module 800, an evaluate 
20 reference updates module 802, an automatic adaptive thresholds module 804, an update 

frame analysis data module 806, and an update motion reference module 808. 

FIGURE 43 is a simplified block diagram of the information processing 

module of FIGURE 16, for evaluating and updating reference images. The evaluate 

reference updates sequence includes a zones motion detection module 822, an update 
25 update counters module 824, a decide standard update module 826, a decide archive 

update module 828, a decide gain change module 830, and a decide learn-outs module 

832. 

FIGURE 44 is a simplified block diagram of the information processing 
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module of FIGURE 16, for changing threshold values relative to changing background 
values from the field of view. The automatic adaptive thresholds sequence includes an 
initialize SL thresholds edge threshold 840, a calculate zone statistics module 842, a 
calculate zone metrics module 844, and an apply metrics module 846. 

5 FIGURE 45 is a simplified block diagram of the information processing 

module of FIGURE 1 6, for determining the geometric association of edge and zone data 
in a detected object. The sequence includes an initialize module 850, an application for 
each qualified edge region 852, an application regarding initialization for a particular 
region 854, and a traverse region bounding rectangle module 856. The sequence then 

10 continues to FIGURE 46. 

FIGURE 46 is a continuation of FIGURE 45 of the simplified block diagram 
of the information processing module of FIGURE 16, and includes testing region edges 
to determine zone intersections in detected objects. The sequence includes a test edge 
region/zone intersection modules 862,864 and a test region/motion/zone intersection 

15 module 866. 

FIGURE 47 is a continuation of FIGURE 46 of the simplified block diagram 
of the information processing module of FIGURE 16, and includes evaluating region 
scores of zones to determine zone intersections in detected objects. The sequence 
includes evaluate region scores modules 872,874, set derived flags update 

20 counter/histories module 876, and an update top detection scores & safety zone latch 

module 878. 

In an embodiment, an automatic door control and safety system is provided 
that controls door behavior in accordance with logic that interprets a nominally optically 
sensed object situation and environment proximate to the door. The system uses a 
25 camera sensor sub-system fitted with an appropriate lens in order to generate an image 

of the desired sensing area. Digital images produced by the camera sub-system are 
processed using image processing in a processing sub-system in order to develop data 
used to drive specific decision logic to effect desired door control. Thus, door control 
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is effected by computer interpretation of image content. 

In an embodiment, from a processing point of view, the system incorporates 
several processing stages: 1) image formation; 2) image conditioning; 3) image 
processing; 4) image content processing; 5) derived data processing; 6) data 
5 interpretation processing; and 7) control logic processing. 

The door control and safety system is supported by hardware elements to 
include the camera sub-system, and a general purpose processor sub-system that can be 
augmented by a digital signal processing device. The camera sub-system can include a 
lens system, a charge-coupled device imaging device, amplifiers, and an analog-to-digital 

1 0 conversion element. These element can be commonly found together in home computer 

applications, for example, which interface a digital camera to produce digital images on 
the computer screen for capture and storage for a variety of purposes. 

The system uses a selection of image processing operators, implemented in 
an algorithm, and subsequent derived data processing and interpretation. The selected 

1 5 image processing operators and image content processing are derived through the optical 

phenomena exhibited by objects within the field of view of the camera. The image 
processing operates on the numbers contained in the array representative of scene 
determined though the lens and camera mounting geometry. This image processing 
creates internal arrays of numbers which are the results of the image processing, to be 

20 used by subsequent operations thus forming a sequence of image processing operations. 

In an embodiment of the system, the entire image field is processed. Furthermore, there 
are no prior assumptions about target objects used to develop any processing elements 
designed to match anticipated object characteristics for the purpose of selecting subsets 
of the entire image field. 

25 At the beginning of the image processing sequence, the image processing 

accepts a new input image of the scene (which is a single time sample ("frame") of the 
on-going image digitization stream). Storage is provided in order to maintain a previous 
image frame for comparison to a newly captured image frame (a "background" image). 
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This stored image frame is captured in the same way as a new frame, and, in particular, 
is a single image frame, not an average of more than one frame. 

In an embodiment, each new image frame is filtered to remove speckle noise 
using a median filter. The median filter removes isolated noise while not blurring the 
5 image as does averaging. Such isolation noise may be due to imaging sensor noise, 

downstream electronics noise or environmentally-produced scintillation. The image 
stored for comparison is filtered one with the median filter, as is the current image. The 
median filter in can be implemented as a 3 x 3 filter kernel that is passed over every pixel 
in the image array. The value at the center of the kernel is deposited in a new image 

10 array, and the value is that which is the median of the nine numbers in the filter kernel. 

After image filtering, two new image arrays are generated (i.e., FIGURES 52 
and 53). The first new image array (FIGURE 52) is determined as the pixel-by -pixel 
difference of the current image minus the background image ("positive contrast"). The 
second new image array (FIGURE 53) is determined as the pixel-by-pixel difference fo 

15 the background image minus the current image ("negative contrast"). The images are 

maintained as arrays of 8-bit numbers, so that when difference values are greater than 
255 or less than 0, values are clipped accordingly. 

After differencing, the images still contain 8-bit values. (Images with 
multiple bit levels are commonly referred to as grey-scale images). After image 

20 differencing, a thresholding operator is applied to each of the resulting positive and 

negative contrast grey-scale images. The threshold values applied to the two images may 
be different. The values can be fixed or adaptive wherein changes are made based on 
downstream image interpretation results. The pixel-by-pixel thresholding operation 
produces two new images. For each image, when the grey level in the input image 

25 exceeds the associated threshold value, a "1" is placed in the output image array, 
otherwise a "0" is placed. The result of the thresholding operation is thus two "binary" 
images. 

Turning to FIGURES 54 and 55, selected binary image processing techniques 
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of mathematical morphology are applied to the binary images to facilitate downstream 
image interpretation. In an embodiment, operators are selected to remove isolated binary 
regions that could not be from significant objects, while improving the "connectedness" 
of larger regions that may be significant. Referred to as shape filtering, each of the two 
5 binary images are filtered similarly to the median filter mechanism (a 3x3 spatial kernel), 
except that the filter kernel operation is a maximum operator followed by a minimum 
operation, not the median operation. Such a filter is referred to as a binary closing or 
"close." A "close" is a "dilation" followed by an "erosion." The "dilation" is the 
maximum operation on the kernel, and the "erosion" is the minimum operation. 

10 Turning to FIGURE 56, the two closed binary images (FIGURES 54 and 55) 

are logically OR-ed pixel-by-pixel to produce a resultant binary image representative of 
both positive and negative contrast differences with respect to the input images. 

Turning to FIGURE 57, a connected components algorithm is applied to the 
resultant binary OR image (FIGURE 56). This algorithm identifies all the connected 

15 binary regions in the image. A connected region is one wherein every member pixel is 

a neighbor of at least one other member pixel. The connected components algorithm 
labels each region and builds a database containing derived features of each region. In 
an embodiment, the features can include region area, bounding rectangle, circularity, 
ellipse major and minor axis lengths, and perimeter. The region feature data is processed 

20 to select regions of interest. The regions are a direct result of the presence of the object 

in the field of view. No operator selection of sub-image regions of the total image field 
is involved in selecting the object-related regions - the regions are determined by the 
object. 

With a database representative of image content, the features of each region 
25 are considered by interpretation logic to develop control logic decisions. In an 

embodiment, the interpretation logic is implemented as a set of "if-then-else" constructs, 
and can utilize arithmetic combination of the basic region features in order to determine 
image content interpretation. For instance, the resulting region area can be used to infer 
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the presence of an object of interest, and the region centroid and bounding rectangle 
determine the location of that object. (The bounding rectangle is the smallest rectangle 
that includes all pixels belonging to the region.) 

In an embodiment, the operator can define rectangular regions of the image 
5 field of view to determine areas for specific control actions. The bounding rectangle 

coordinates of the computer-derived object regions of interest are compared to the 
coordinates of the operator-determined decision regions in order to determine subsequent 
control logic results. If an object is declared to be in the safety zone, for example, the 
control logic indicates that the door should remain open until the safety zone is clear. 

10 Similarly, if an object is determined to be in the activation zone (the binary region 

bounding rectangle representative of the image object intersects the activation zone 
decision rectangle), then the signal is sent to open the door. In an embodiment, the image 
regions selected by the operator for control logic purposes are not used in any way to 
initialize or otherwise influence the image processing of the entire image in order to 

15 determine image content. 

While the specific embodiments have been illustrated and described, 
numerous modifications come to mind without significantly departing from the spirit of 
the invention, and the scope of protection is only limited by the scope of the 
accompanying Claims. 
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CLAIMS 

1 . A detector comprising : 

a sensor for collecting electromagnetic energy from a field of view for the 
sensor, die sensor generating a signal indicative of objects in die field of view; 
5 an image analyzer electronically coupled to the sensor for receiving the sensor 

signal and forming an image of pixels therefrom; 

an edge detector adapted to detect an edge of an object depicted within the 
pixel image, and determining the position of the edge within the pixel image; 

a data storage device for storing edge location data of a first image; and 
1 o a comparator for comparing edge detection data from a second image to edge 

detection data from the first image in order to determine if the object associated with the 
edge has moved since the acquisition of the first image. 

2. A shadow and light beam detection process comprising the steps of: 
capturing a grey scale image; 

1 5 generating a binaiy image to define a binary map of the grey scale image; and 

comparing the binary map to a copy of the grey scale image to determine grey 
scale values corresponding to selected portions of the binary map. 

3. A process for determining whether a discrete area appearing in image 
information for a field of view represents a shadow or a light beam anomaly comprising: 

20 predetermining a characteristic of the anomaly; 

predetermining a calculation that will quantify the characteristic; 
applying the calculation to the image information and generating a quantity 
of the characteristic; and 

classifying the area based upon the quantity calculated. 
25 4. A process for determining an edge comprising: 

predetermining a threshold value to apply to a grey scale image value in a first 
and second channel; 

comparing the image value to the threshold value in each of the first and 
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second channel; 

outputting a first binary value in each of the first and second channel; 

defining a first dilation pixel having the first binary value; 

defining a plurality of dilation pixels adjacent the first dilation pixel, each of 
the plurality of dilation pixels having a second binary value; 

comparing the first binary value of the first dilation pixel and the second 
binary value of each of the plurality of dilation pixels; 

outputting a dilated binary value in each of the first and second channel; and 

oring the dilated binary value in the first channel and the dilated binary value 
in the second channel to generate an ored dilated binary value. 
5. A process for determining an edge comprising: 

storing a first grey scale image in a first buffer; 

reading the first image; 

transferring the first image to a second buffer; 

repeating the steps of storing, reading and transferring for a second grey scale 

image; 

subtracting the first image from the second image to generate a first positive 

image; 

subtracting the second image from the first image to generate a second 
positive image; 

comparing the first and second positive image to generate a positive 
difference image; 

predetermining a threshold value to apply to the positive difference image in 
a first and second channel; 

comparing the positive difference image to the threshold value in each of the 
first and second channel; and 

outputting a first binary value in each of the first and second channel. 
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6. A motion detection process comprising the steps of: 
capturing a first image; 

detei-mining edges of objects extant in the first image; 
capturing a second image; 
5 determining edges of objects extant in the second image; 

subtracting the first image from the second image to generate a difference 

image; and 

comparing at least one of the edge determinations to the difference image to 
determine movement of an object in the field of view represented by the at least one 
10 edge. 

7. An image proces s for determining camera problems comprising the steps of: 
capturing an image; and 

creating a histogram to evaluate the image against pre-selected criteria. 

8. An image process for determining camera problems comprising the steps of: 
15 capturing a first image; 

detennining edges of the first image; 
capturing a second image; 
determining edges of the second image; 

subtracting the second image from the first image to generate a difference 

20 image; and 

thresholding the difference image. 

9. An image process comprising the steps of: 
capturing an image; 

analyzing the image to define objects in the image; 
25 labeling objects in the image; and 

translating the objects to real world coordinates. 

10. A process for determining an edge of an object in a field of view of a camera 
recording pixel data comprising: 
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capturing a reference frame of pixel data; 
capturing an image frame of pixel data; 

subtracting the reference frame from the image frame to generate a first gray 
scale contrast frame; 

5 subtracting the image frame from the reference frame to generate a second 

gray scale contrast frame; 

thresholding the first and second gray scale contrast frame to generate first 
and second contrast threshold frames; 

dilating the first and second contrast threshold frames to generate first and 
10 second dilated frames; 

oring the first and second dilated frames; and 

adjusting the first and second dilated frames to generate a first and second 
binary images. 

11. A process for smoothing pixel data defined as an edge of an obj ect determined 
15 to exist in an image comprised of the pixel data, the process comprising the steps of: 

defining a grey scale image from the pixel data in a first and second channel; 

filtering the grey scale image utilizing a horizontal Sobel kernel to generate 
a horizontal filtered output in the first and second channel; 

filtering the grey scale image utilizing a vertical Sobel kernel to generate a 
20 vertical filtered output in the first and second channel; 

deteraiining the absolute values of the horizontal and vertical filtered output 
in the first and second channel; 

summing the absolute values to generate a Sobel value in the first and second 
channel; and 

25 subtracting the Sobel value in the first channel from the Sobel value in the 

second channel to generate an output value. 

12. A system for analyzing an image comprising: 

a sensor configured and adapted to sense one of either active or passive 
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energy or both from a field of view; 

an imager configured to form an image from the energy sensed by the sensor; 

a processor responsive to the imager, the processor being configured and 
adapted to label image content information from the image; and 
5 a decision maker responsive to the processor, the decision maker being 

configured and adapted to make determinations about the presence of a/an (stationary) 
object in the field of view from the image content information. 

1 3 . The system of claim 1 2 wherein the decision maker having a decision metric, 
the decision metric being based upon the relationship of selected image information 

10 content to a first predetermined area or zone within the image. 

1 4 . The system of claim 1 3 wherein the decision maker having a decision metric, 
the decision metric being based upon the relationship of selected image information 
content to a second predetermined area or zone within the image. 

1 5 . The system of claim 1 3 or 1 4 wherein the selected image information content 
15 is determined by the decision maker to be indicative of the object. 

16. The system of claim 1 5 wherein the relationship to the area or zone is limited 
to whether at least a portion of the object is within the first or second predetermined 
zones. 

1 7 . The system of claim 1 3 or 1 4 wherein the selected image information content 
20 is determined by the decision maker to be indicative of an edge of the object. 

1 8. The system of claim 13 or 14 wherein the selected image information content 
is determined by the decision maker to be indicative of a shadow of the object. 

19. The system of claim 13 or 14 wherein the selected image information content 
is determined by the decision maker to be indicative of a lightbeam. 

25 20. The system of claim 13 or 14 wherein the selected image information content 

is determined by the decision maker to be indicative of motion of the object. 
21 . The system of claim 12 wherein the processor is configured and adapted to 

determine features of the object selected from the group consisting of area, perimeter, 
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centroid, bounding rectangle, magnitude, variance and shape. 



22. The system of claim 21 wherein the shape is selected from the group 
consisting of elongation, spread, ellipse shape factor and orientation. 

23 . A method of image analysis by a computing device comprising the steps of: 
5 providing to the device, a pixel image of at least a portion of a field of view 



of a sensor; 

the device determining pixels in the image which likely correspond to at least 
one edge of at least one object within the field of view; and, 

the device labeling at least one group of the pixels identified to define an 
10 individual entity representative of the at least one edge of the at least one object in the 

field of view. 

24. The method of claim 23 including the step of the device mathematically 

operating on the at least one object or the at least one group of pixels to generate at least 
one feature of the at least one object or the at least one group of pixels. 
15 25. The method of claim 24 including the step of the device mathematically 

operating on the at least one object or the at least one group of pixels to generate a 
mathematical or geometric feature of the at least one object or the at least one group of 
pixels. 

26. The method of claim 24 wherein the generating at least one feature step 
20 includes determining one or more features of the at least one object selected from the 

group consisting of area, perimeter, moments of inertia, grey level mean, grey level 
variance, grey level histogram and location. 

27. A method of image analysis by a computing device comprising the steps of: 
providing to the device, a pixel image of at least a portion of a field of view 

25 of a sensor; 

the device quantifying multiple properties of all pixels in an area of interest; 
the device labeling a first group of the pixels so as to satisfy a predetermined 
first criterion representative of an object in the field of view; and, 
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the device independently and simultaneously labeling a second group of the 
pixels so as to satisfy a predetermined second criterion representative of the object in the 
field of view. 

28. The method of claim 27 including the step of the device mathematically 
5 operating on the object or the first and second group of pixels to generate at least one 

feature of the object or the first and second groups of pixels, respectively. 

29. The method of claim 28 including the step of the device mathematically 
operating on the object or the first and second group of pixels to generate a mathematical 
or geometric feature of the object or the first and second groups of pixels, respectively. 

10 30. The method of claim 28 wherein the generating at least one feature step 

includes determining one or more features of the object or the first and second groups of 
pixels selected from the group consisting of area, perimeter, moments of inertia, grey 
level mean, grey level variance, grey level histogram and location. 

3 1 . The method of claim 27 wherein the device performs the steps of: 

15 mathematically operating on the object or the first and second groups of 

pixels to generate at least two features of the object or the first and second groups of 
pixels, wherein at least one feature is based on edge data and at least one feature is based 
on region analysis; 

comparing the features of the object to a predefined set of rules stored in or 
20 accessible to the device; and, 

the device generating a decision with respect to controlling a second device, 
based upon the comparison. 

32. A method of image analysis comprising the steps of: 
generating a time sequence of images in a field of view; 

25 allocating one of the successive images, other than an immediately preceding 

image, in the field of view as a reference image; and, 

comparing selected successive images to the reference image of the same field 

of view. 
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33 . The method of claim 32 including the step of refreshing the reference image 
from time to time based on a criteria. 

34. The method of claim 33 including the step of refreshing the reference image 
based upon evaluation of image content derived from the image comparisons. 

5 35. The method of claim 3 3 including the step of refreshing the reference frame 

based upon a predetermined passage of time. 

36. The method of claim 32 including the step of updating a portion of the 
reference image using a current image. 

37. The method of claim 36 including the step of performing an update using 
10 predetermined criteria and evaluation of image content. 

38. A method of image analysis comprising the steps of: 
generating a time sequence of images hi a field of view; 

allocating one of the successive images in the field of view as a reference 
image; and, 

1 5 comparing selected successive images to the reference image of the same field 

of view using a positive difference operator. 

39. A method of image analysis comprising the steps of: 
capturing an image; 

deriving one or more images from the captured image using predetermined 
20 image operators; 

labeling individual groups of pixels independently in one or more of the 
derived images corresponding to objects in the field of view; and 

calculating features by evaluating the same groups of pixels for each of the 
one or more derived images independently. 
25 40. The method of claim 39 including the step of considering the features 

simultaneously. 

41 . The method of claim 39 including the step of basing one of the features on 

edge data and another of the features on region data. 
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42. A camera system for controlling a device other than a camera comprising: 
a camera having a lens, the lens having at least a first and second radial 

distortion coefficient, the first coefficient having a value of at least about 1 .0 and the 
second coefficient having an absolute value of at least about 1 .0; 
5 a signal generator for generating a camera signal indicative of a field of view 

through the lens; 

an analyzer which receives the camera signal and analyzes the camera signal 
for detection of an object or motion, the analyzer generating a detection signal indicating 
the object or motion; and, 
10 a device controller receiving the detection signal and controlling the device 

in response thereto. 

43. The system of claim 42 wherein the analyzer having means for analyzing 
image content while compensating for the radial distortion. 

44. The system of claim 43 wherein the analyzer determines the radial distortion. 
15 45 . The system of claim 42 wherein the analyzer having means for determining 

the size of objects in the field of view. 

46. The system of claim 42 including code for representing radial distortion as 

a standard radial distortion polynomial system wherein the polynomial system having 
sixth-powered terms, the coefficients of which are non-zero. 
20 47. A camera system for controlling a device other than a camera comprising: 

a camera having a lens, the lens having a field of view of at least about 53 

degrees; 

a signal generator for generating a camera signal indicative of the field of 
view through the lens; 

25 an analyzer which receives the camera signal and analyzes the camera signal 

for detection of an object or motion, the analyzer generating a detection signal indicating 
the object or motion; and, 

a device controller receiving the detection signal and controlling the device 
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in response thereto. 

48. The system of claim 47 wherein the lens having a field of view of at least 90 
degrees. 

49. The system of claim 47 wherein the analyzer having means for analyzing 
5 image content while compensating for the radial distortion. 

5 0 . The system of claim 49 wherein the analyzer determines the radial distortion. 

5 1 . The system of claim 47 wherein the analyzer having means for determining 
the size of objects in the field of view. 

52. The system of claim 47 including code for representing radial distortion as 
10 a standard radial distortion polynomial system wherein the polynomial system having 

sixth-powered terms, the coefficients of which are non-zero. 

53. A camera system for controlling a device other than a camera comprising: 
a camera having a lens, the lens having an image diagonal and a focal length, 

wherein the image diagonal is greater than the focal length; 
15 a signal generator for generating a camera signal indicative of the field of 

view through the lens; 

an analyzer which receives the camera signal and analyzes the camera signal 

for detection of an object or motion, the analyzer generating a detection signal indicating 

the object or motion; and, 
20 a device, controller receiving the detection signal and controlling the device 

in response thereto. 

54. The system of claim 53 wherein the image diagonal is at least twice the focal 
length. 

55. The system of claim 53 wherein the analyzer having means for analyzing 
25 image content while compensating for the radial distortion. 

5 6 . The system of claim 5 5 wherein the analyzer determines the radial distortion. 

57. The system of claim 53 wherein the analyzer having means for determining 

the size of objects in the field of view. 
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58. The system of claim 53 including code for representing radial distortion as 
a standard radial distortion polynomial system wherein the polynomial system having 
sixth-powered tenns, the coefficients of which are non-zero. 

59. A camera comprising : 
5 a) a lens; 

b) a detector located so as to have light focused upon the detector from the 
lens, the detector generating an image signal; 

c) a controller receiving the image signal; and 

d) a gain control adjusting gain of the image signal from the detector based 
10 upon a histogram of grey-scale values of the image. 

60. The camera of claim 59 wherein the detector is selected from the group 
consisting of a CCD-based detector and a CMOS-based detector. 

61. A method of image processing comprising the steps of: 

capturing a pixel image of a field of view from a single point of view; 
15 electronically analyzing the image to determine pixel groups in the image 

corresponding to objects in the field of view; and 

determining the spatial location of objects in the field of view corresponding 
to the pixel groups. 

62. A method of image processing comprising the steps of: 

20 providing an array of sensors, the array having a first field of view of a 

volumetric space; 

defining a two-dimensional pixel image from signals sent from the array; (the 
pixel image providing a two-dimensional representation of the volumetric space) 

predetermining a desired coordinate system in the volumetric space; and, 
25 utilizing information regarding a pose of the array and the coordinate system 

to electronically determine the relation between pixel coordinates in the image to spatial 
coordinates in the three-dimensional space. 

63 . The method of claim 62 including the steps of: 
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providing a lens with a radial distortion to focus energy from a second field 
of view to the array; and, 

adjusting the relation of pixel values to spatial location. 

64. The method of claim 62 including the step of electronically determining the 
5 effective resolution of the array of sensors within the field of view relative to the 

coordinate system unit of measure. 

65. The method of claim 64 including the step of controlling a detection 
characteristic over the entire field of view by utilizing the determined resolution values. 

66. The method of claim 64 including the step of adjusting detection threshold 
10 limits by utilizing the determined resolution values. 

67 . The methods of claims 64-66 including the step of predetermining a pixel area 
corresponding to a representative size of objects of interest for detection. 

68. The method of claim 62 including the step of providing an array of sensors 
arranged in a substantially planar relationship. 

15 69 . The method of claim 62 including the step of utilizing information regarding 

location and orientation of the array and the coordinate system to electronically determine 
the relation between pixel coordinates in the image to spatial coordinates in the three- 
dimensional space. 

70 . The method of claim 62 including the step of utilizing information regarding 
20 intrinsic parameters of the imaging system and the coordinate system to electronically 

determine the relation between pixel coordinates in the image to spatial coordinates in 
the three-dimensional space. 

71 . The method of claim 62 including the step of predetermining a homogeneous 
coordinate system in the volumetric space. 

25 72. The method of claim 71 including the step of utilizing information regarding 

a pose of the array and the homogeneous coordinate system to electronically determine 
the relation between pixel coordinates in the image to spatial coordinates in the three- 
dimensional space. 
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73 . The method of claim 7 1 including the step of utilizing a proj ective perspective 
transformation to electronically determine the relation between pixel coordinates in the 
image to spatial coordinates in the three-dimensional space. 

74. The method of claim 73 including the step of adding a radial distortion 
5 correction to the projective perspective transformation to electronically determine the 

relation between pixel coordinates in the image to spatial coordinates in the three- 
dimensional space. 

75. The method of claim 62 including the step of providing a CCD, the CCD 
having a first field of view of a volumetric space. 

10 76. A method of image analysis comprising the steps of: 

providing a pixel image of at least a portion of a field of view of a sensor; 
determining pixels in the image which likely correspond to objects in the field 

of view; 

discriminating between pixels so determined to define spatially discrete 
15 groups of pixels; and, 

labeling each group of pixels. 

77. A timer for monitoring a processor comprising: 

a first circuit for resetting the processor when improper function is detected; 

and 

20 a second circuit for maintaining a safe output condition when the processor 

is not controlling the output condition. 

78. The timer of claim 77 including a temperature monitor. 

79. The timer of claim 78 wherein the temperature monitor generates an output 
signal when the processor temperature exceeds predetermined operating limits. 

25 80. The timer of claim 79 wherein the temperature monitor is connected to the 

first circuit for resetting the processor when the processor temperature exceeds the 
predetermined operating limits. 
81. A monitoring device comprising: 
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a sensor for sensing energy from an environment and providing an output 
indicative of same; 

an image formed from the output of the sensor; 

a first analyzer for analyzing a first portion of the image; and 
5 a second analyzer for analyzing a second portion of the image distinct from 

the first portion of the image and each generating a signal indicative of the respective 
portion of the image analyzed by the respective analyzer. 

82. A monitoring device comprising: 

a sensor for sensing energy from an environment and providing an output 
10 indicative of same; 

an image formed from the output of the sensor; 

n analyzers for analyzing n portions of the image, each portion being distinct; 

and 

each analyzer generating a signal indicative of the respective portion of the 
15 image analyzed by the respective analyzer. 

83 . A camera having a lens and optionally a transparent lens cover and having a 
field of view through the lens and optionally through the lens cover comprising: 

a storage device for storing logic and data and codifying criteria representative 
of conditions having an effect on the field of view; 
20 an imager for obtaining and storing an image representative of the field of 

view; 

a processor for processing data derived from the image for use in comparison 
to the criteria; and 

a generator for generating control signals representative of the image. 
25 84. The camera of claim 83 including frame analysis algorithms for evaluating 

image quality of the camera. 

85. The camera of claim 83 including frame analysis algorithms for evaluating 

changes in position or orientation of the lens relative to camera environment. 
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86. The camera of claim 83 including frame analysis algorithms for evaluating 
image failure of the camera. 

87. The camera of claim 83 wherein the conditions that impact the view through 
the lens and optionally a lens cover are selected from the group consisting of substances 

5 contacting the lens or lens cover, level of lighting in the view, unusually bright or dark 

spots in the view, overall flux of electromagnetic energy passing through the lens, and 
obstruction of the lens by other objects. 

8 8 . The camera of claim 83 wherein the processor has a generator for generating 

histograms from the image and the storage device has data relative to histogram criteria. 
10 89. The camera of claim 88 wherein the data relative to histogram criteria is 

derived by a comparison between a predetermined histogram and a histogram determined 
from the image representative of the field of view. 

90. The camera of claim 87 wherein the processor has a generator for generating 

histograms from the image and the storage device has data relative to histogram criteria. 
15 91. The camera of claim 90 wherein the data relative to histogram criteria is 

derived by a comparison between a predetennined histogram and a histogram determined 
from the image representative of the field of view. 

92. The camera of claims 87-91 wherein the substances contacting the lens or lens 
cover are selected from the group consisting of dirt, dust, oil, water, organic film, 

20 inorganic film, soot and metal fines. 

93 . The camera of claim 83 wherein the conditions are selected from the group 
consisting of normal and abnormal. 

94. A system for defogging a window comprising: 
an electrically conductive terminal; and 

25 a PTC material adj oining the conductive terminal and operably coupled to the 

window. 

95. The system of claim 94 further comprising another electrically conductive 
terminal adjoining the PTC material. 
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96. The system of claim 95 further comprising a voltage potential coupled to the 
electrically conductive terminals. 

97. The system of claim 94 wherein the electrically conductive terminal has an 
aperture extending therethrough. 

5 98 . The system of claim 94 wherein the PTC material has an aperture extending 

therethrough. 

99. The system of claim 94 wherein the electrically conductive terminal and the 

PTC material have apertures extending therethrough and in coaxial alignment with each 
other. 

10 100. An apparatus comprising: 

a housing having an opening; 

a window mounted over the opening; 

a PTC material operably coupled to the window. 

101. The apparatus of claim 100 further comprising an electrically conductive 
1 5 terminal connected to the PTC material. 

1 02. The apparatus of claim 101 further comprising another electrically conductive 
terminal adjoining the PTC material. 

103. The apparatus of claim 1 0 1 further comprising a voltage potential coupled to 
the electrically conductive terminal. 

20 1 04. The apparatus of claim 1 0 1 wherein the electrically conductive terminal has 

an aperture extending therethrough. 

105. The apparatus of claim 100 wherein the PTC material has an aperture 
extending therethrough. 

106. The apparatus of claim 101 wherein the electrically conductive terminal and 
25 the PTC material have apertures extending therethrough and in coaxial alignment with 

each other and the opening in the housing. 

107. The apparatus of claim 1 00 further comprising a camera mounted within the 
housing. 
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108. The apparatus of claim 1 07 wherein the camera has a field-of-view extending 
through the window. 

1 09. The apparatus of claim 1 08 wherein the field-of-view includes at least part 
of a pathway to a door. 

5 110. The apparatus of claim 109 further comprising a door opener operably 

coupled to the door and responsive to objects detected within the field-of-view. 

111. An input device for communicating with a controller for an automatic door 
comprising: 

a first sequencing key which is configured to prompt a user to enter a first set 
10 of data into the device when actuated a first time and prompt the user to enter a second 

set of data when actuated a second time; 
at least one input key; and 
at least one input. 

112. The device of claim 111 including a display for displaying user input. 

15 113. The device of claim 1 1 1 including a display for displaying pre-stored user 

options for selection in response to prompts from the first sequencing key. 

114. The device of claim 113 wherein the display displaying only a portion of the 
pre-stored data to be displayed for each actuation of the at least one input key. 

115. The device of claims 111-114 including an input selector which accepts and 
20 stores user input when actuated. 

1 16. The device of claims 1 12-1 15 including a second sequencing key which is 
configured to prompt a user to enter a third set of data into the device using a first display 
key when actuated a first time and prompt the user to enter a fourth set of data into the 
device using the first display key when actuated a second time. 

25 117. The device of claims 1 12-1 15 including a second sequencing key which is 

configured to prompt a user to enter a third set of data into the device using the first 
display key when actuated a first time and prompt the user to enter a fourth set of data 
into the device using a second display key when actuated a second time. 
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118. The device of claims 112-117 including a third sequencing key which is 
configured to prompt a user to enter a fifth set of data into the device using a first display 
key when actuated a first time and prompt the user to enter a sixth set of data into the 
device using the first display key when actuated a second time. 
5 119. The device of claims 112-117 including a third sequencing key which is 

configured to prompt a user to enter a fifth set of data into the device using the first 
display key when actuated a first time and prompt the user to enter a sixth set of data into 
the device using a second display key when actuated a second time. 

120. The device of claim 116 wherein the first and second sequencing keys are 
10 configured with a hierarchy, the hierarchy allowing the first sequencing key to override 

operation of the second sequencing key. 

121. The device of claim 118 wherein the first and third sequencing keys are 
configured with a hierarchy, the hierarchy allowing the first sequencing key to override 
operation of the third sequencing key. 

15 1 22 . The device of claim 1 1 8 wherein the first, second and third sequencing keys 

are configured with a hierarchy, the hierarchy allowing the first sequencing key to 
override operation of the second and third sequencing keys. 

123 . The device of claim 111 wherein the at least one input is a numerical input. 

124. The device of claim 1 1 1 wherein the at least one input is an alpha input. 

20 125. A sensor system for controlling an automatic door which has a door panel 

selectively blocking an opening comprising: 

a sensor having a field of view of areas of interest about the opening and a 
signal output relative to objects sensed in the field of view of the sensor; 
a signal processor responsive to sensor output signals; 
25 a door drive responsive to the signal processor; and 

an input device having a signal output, the signal processor responsive to 
output signals from the input device, 

wherein the input device having a pose input for permitting a user to input 
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data indicative of the pose of the sensor as mounted to obtain the field of view. 

126. The sensor system of claim 125 wherein the pose input includes installed 
height of the sensor. 

127. The sensor system of claim 125 wherein the pose input includes a nominal 
5 height of the sensor. 

128. The sensor system of claim 125 wherein the pose input includes a model 
number of the door associated with the field of view. 

129. . The sensor system of claim 125 wherein the pose input includes a width of 
the opening. 

10 130. The sensor system of claim 125 wherein the pose input includes a length of 

the opening. 

131. The sensor system of claim 125 wherein the pose input includes a first 
position of an activation zone. 

132. The sensor system of claim 125 wherein the pose input includes a second 
15 position of an activation zone. 

133. An input device for communicating with a door drive responsive to a sensor 
which has a field of view of areas of interest about the door, the device comprising: 

a first tier of selectable input selectors; and 
a second tier of selectable input selectors; 
20 wherein the first and second tiers being configured such that user input into 

the first tier of selectable input selectors controls the availability of selections for input 
on the second tier of selectable input selectors. 

134. A sensor system for controlling an automatic door which has a door panel 
selectively blocking an opening, the sensor system comprising: 

25 a sensor configured to sense objects in a field of view; and 

an input device having an output for communication with a controller for the 
automatic door, the input device having a user input relative to the sensor height. 

135. The system of claim 1 34 wherein the input device having a user input relative 
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to the door width. 

136. The system of claim 134 wherein the input device having a data set stored 
relative to door model. 

137. An image analyzer for data processing comprising: 
5 a FPGA for pixel processing; 

a CPU operably comiected to the FPGA in parallel; and 
a video buffer operably connected to the CPU. 

138. A process for performing high-rate data processing comprising the steps of: 
providing a CPU for processing a current frame; 

10 providing a FPGA for simultaneously processing a next frame to generate a 

FPGA output; 

storing the FPGA output in a storage bank; 
retrieving the FPGA output from the storage bank; and 
sending the FPGA output to the CPU for processing. 
15 139. A process for initializing a video system comprising the steps of: 
initializing a FPGA; 
selecting a first ping-pong data set; 
instructing the FPGA to capture a video frame; 
initializing at least one reference image; 
20 instructing the FPGA to process the at least one reference image to generate 

FPGA outputs; and 

transferring the FPGA outputs from the FPGA to the first ping-pong data set. 
140. A system for controlling an effector comprising: 

a sensor configured and adapted to sense one of either active or passive 
25 energy or both from a field of view; 

an imager configured to form an image from the energy sensed by the sensor; 
an image analyzer responsive to the imager, the image analyzer being 
configured and adapted to define image content information from the image; 
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a decision maker responsive to the image analyzer, the decision maker being 
adapted and configured to make determinations about the presence of a/an (stationary) 
object in the field of view from the image content information; and, 

a first controller for controlling the effector, the first controller being 
5 responsive to the decision maker. 

141 . The system of claim 140 wherein the decision maker having a first decision 
metric, the first decision metric being based upon the relationship of selected image 
information content to a first predetermined area or zone within the image. 

142. The system of claim 141 wherein the decision maker having a first decision 
10 metric, the first decision metric being based upon the relationship of selected image 

information content to a second predetermined area or zone within the image. 

143. The system of claim 141 or 142 wherein the selected image information 
content is determined by the decision maker to be indicative of the object. 

144. The system of claim 143 wherein the relationship to the area or zone is 
15 limited to whether at least a portion of the object is within the first or second 

predetermined zones. 

145. A system for controlling an automatic door which selectively blocks an 
opening, the system comprising: 

a sensor configured and adapted to sense one of either active or passive 
20 energy or both from a field of view; 

an imager configured to form an image from the energy sensed by the sensor; 

an image analyzer responsive to the imager, the image analyzer being 
configured and adapted to define image content information from the image; 

a decision maker responsive to the image analyzer, the decision maker being 
25 adapted and configured to make determinations about the objects in the field of view 

based upon the image content information; and, 

a door controller for controlling at least the opening and closing of the door, 
the door controller being responsive to the decision maker. 
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146. The system of claim 145 wherein the decision maker having a first decision 
metric, the first decision metric being based upon the relationship of selected image 
information content to a first predetermined area or zone within the image. 

147. The system of claim 146 wherein the decision maker having a first decision 
5 metric, the first decision metric being based upon the relationship of selected image 

information content to a second predetermined area or zone within the image. 

148. The system of claim 146 or 147 wherein the selected image information 
content is indicative of the presence of a stationary object. 

149. The system of claim 1 48 wherein the relationship to the area or zone includes 
10 whether at least a portion of the object is within the first or second predetermined zones. 

150. The system of claim 1 46 or 1 47 wherein the relationship to the area or zone 
includes whether at least a portion of an obj ect is within the first or second predetermined 
zones. 

151. The system of claim 145 including a geometry mapper which translates the 
15 dimensions and geometries of the field of view to coordinates of pixels comprising the 

images. 

1 52. An object position locator consisting of a single camera for generating pixel 
images for analysis, comprising: 

an image analyzer having an object detection algorithm which determines the 
20 existence of objects in a field of view of the camera; 

a labeler for providing a discrete identity to an object in the pixel image; and 
a position locator for determining coordinates of the object within the pixel 

image. 

153. A monitoring device comprising: v 

25 a sensor for sensing energy from an environment and providing an output 

indicative of same; 

an image formed from the output of the sensor; 

n analyzers for analyzing n portions of the image, each portion being distinct; 
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and 

each analyzer generating a signal indicative of the respective portion of the 
image analyzed by the respective analyzer. 

1 54. A surveillance system comprising: 

5 multiple sensors, each sensor being deployed at an individual location of 

interest, each sensor having a field of view and generating a sensor signal indicative of 
objects within its field of view; 

one or more signal analyzers for receiving sensor signals and determining 
from the signals the presence of objects of interest or motion in the respective fields of 
10 view of the multiple sensors, and for generating a display signal representative of the 

multiple fields of view and for generating detection signals indicative of object presence 
or motion in respective ones of the multiple fields of view; 

at least one viewer for receiving display signals and generating and displaying 
human interpretable data representative of selected fields of view; and 
1 5 a controller receiving at least the detection signals from the signal analyzers, 

the controller determining from the detection signals which field of view will be 
displayed on the at least one viewer. 

155. A system for defogging a window comprising: 
an electrically conductive terminal; and 

20 a PTC material adj oining the conductive terminal and operably coupled to the 

window. 

1 56. The system of claim 1 55 further comprising another electrically conductive 
terminal adjoining the PTC material. 

157. The system of claim 156 further comprising a voltage potential coupled to the 
25 electrically conductive terminals. 

158. The system of claim 1 55 wherein the electrically conductive terminal has an 
aperture extending therethrough. 

159. The system of claim 1 5 5 wherein the PTC material has an aperture extending 
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therethrough. 

160. The system of claim 155 wherein the electrically conductive terminal and the 
PTC material have apertures extending therethrough and in coaxial alignment with each 
other. 

5 161. An apparatus comprising: 

a housing having an opening; 

a window mounted over the opening; 

a PTC material operably coupled to the window. 

162. The apparatus of claim 161 further comprising an electrically conductive 
10 terminal connected to the PTC material. 

163. The apparatus of claim 1 62 further comprising another electrically conductive 
terminal adjoining the PTC material. 

1 64. The apparatus of claim 1 62 further comprising a voltage potential coupled to 
the electrically conductive terminal. 

15 165. The apparatus of claim 1 62 wherein the electrically conductive terminal has 

an aperture extending therethrough. 

166. The apparatus of claim 161 wherein the PTC material has an aperture 
extending therethrough. 

1 67. The apparatus of claim 1 62 wherein the electrically conductive terminal and 
20 the PTC material have apertures extending therethrough and in coaxial alignment with 

each other and the opening in the housing. 

168. The apparatus of claim 1 6 1 further comprising a camera mounted within the 
housing. 

169. The apparatus of claim 168 wherein the camera has a field-of-view extending 
25 through the window. 

170. The apparatus of claim 169 wherein the field-of-view includes at least part 
of a pathway to a door. 

171. The apparatus of claim 170 further comprising a door opener operably 
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coupled to the door and responsive to objects detected within the field-of-view. 

172. A system for controlling a door comprising: 
a camera for collecting image data; 

a control unit receiving image data from the camera; and 
5 a drive motor for controlling the opening and closing of the door, the drive 

motor receiving control signals from the control unit. 

1 73 . The system of claim 1 72 wherein the control unit has means for defining at 
least a portion of an image as a control zone. 

1 74. The system of claim 1 73 including a first control zone wherein selected image 
10 data within the first control zone will cause a control signal to open the door. 

175. The system of claim 173-174 including a second control zone wherein 
selected image data within the second control zone will generate a first control condition ' 
preventing the door from closing. 

176. The system of claim 173 wherein the means for defining includes choosing 
15 coordinates from all pixel coordinates by direct access within the control zone. 

177. The system of claim 173 wherein the means for defining includes choosing 
from multiple predefined zones. 

178. The system of claim 173 wherein the means for defining includes putting real 
objects in a field of view so as to delineate boundary coordinates and the real objects 

20 become part of the image data, the control zone being defined from the real obj ects image 

data. 

179. The system of claim 172-174 wherein 

the control unit analyzes the image data in the first control zone at a first time; 
the control unit analyzes the image data in the first control zone at a second 

25 time; and 

the control unit compares the image data at the first time and the image data 
at the second time to determine whether an object is present in the first control zone. 
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1 80. The system of claim 179 wherein the object is analyzed to determine if any 
portion of the object is within the first control zone. 

181. The system of claim 180 wherein the selected image data within the first 
control zone causes the control signal to open the door if any portion of the object is 

5 within the first control zone. 

182. The system of claim 181 wherein the control unit chooses a portion of the 
object to determine a bottom edge of the object. 

1 83 . The system of claim 1 82 wherein the control unit analyzes the bottom edge 
to determine if the bottom edge is within the first control zone. 

10 184. The system of claim 183 wherein a fourth control zone is defined adj acent the 

first control zone, the fourth control zone possessing objects between the first control 
zone and the camera. 

185. The system of claim 184 wherein the fourth control zone is positioned 
between the first control zone and the camera. 
15 186. The system of claim 185 wherein the presence of the object in the fourth 

control zone will prevent the first control zone from sending the control signal to open 
the door. 

187. The system of claim 186 wherein the presence of the object in the fourth 
control zone will cause the first control zone to send the control signal to open the door. 
20 1 88. The system of claim 172-175 wherein 

the control unit analyzes the image data in the second control zone at a first 

time; 

the control unit analyzes the image data in the second control zone at a second 

time; and 

25 the control unit compares the image data at the first time and the image data 

at the second time to determine whether an object is present in the first control zone. 
1 89. The system of claim 172-175 further including a third control zone wherein 
selected image data within the third control zone will generate a second control condition 
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preventing the door from closing. 

190. The system of claim 172-175 wherein the second control zone comprises a 
plurality of predeteimined shapes. 

191. The system of claim 189-190 wherein the third control zone comprises a 
5 plurality of predetermined shapes, the shapes of the third control zone being 

complementary to the shapes of the second control zone. 

1 92 . The system of claim 1 9 1 wherein 

the control unit analyzes the image data in the third control zone at a first 

time; 

1 0 the control unit analyzes the image data in the third control zone at a second 

time; and 

the control unit compares the image data at the first time and the image data 
at the second time to determine whether an object is present in the third control zone. 

1 93 . The system of claim 172-174 wherein the first control zone includes means 
15 for comparing the image data to a plurality of user specified dimensions. 

194. The system of claims 172-193 wherein the camera senses energy selected 
from the group consisting of: visible light waves, infrared lights waves, microwaves, 
radar and sound waves. 

195. The system of claim 173-194 wherein a user can enter data to define at least 
20 one control zone parameter from the group consisting of area, location, shape, number 

of control zones and control criteria. 

1 96. The system of claims 1 -1 94 wherein the sensor is a camera. 

197. A system for controlling a door comprising: 

a sensor, the sensor having, for example, a beam for sensing coherent energy 
25 from objects near the door; 

a control unit receiving electronic values from the sensor indicative of energy 
sensed by the camera; 

a drive motor for controlling the opening and closing of the door, the drive 
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motor receiving control signals from the control unit; and 

means for defining a portion of a beam pattern as a control zone. 
198. The system of claim 1 97 wherein the control zone is a beam pattern 
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Image Operator Definitions 

(-) Positive difference (of two grey-scale (8-bH) images) 

Pij-Qij = Pij-QrfifPij>Qij 
= 0, otherwise. 

+ pixel-wise (bit-wise) (inclusive) logical OR (of two binary images) 

T threshold (an 8-bit image) 

lfPij>THRESthen 
Pij = 1 

Else 

Pij = 0 

IT Invertrthreshold 

Pi) = 255-Pij 
lfPO>THRESthen 
P«-1 

Else 

P«j = 0 

E binary erosion, 3x3 

3x3 kernel Ke 



1 


1 


1 


1 


c 


1 


1 


1 


1 



if center = 0, do nothing, next kernel position 
rf center = 1 {black} then test to see if should change to white. . . 
if sum of 8 neighbors <= 4, set center = 0 

set image boundary to 0 

D binary dilation, 3x3 

3x3 kernel Kd 
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1 


1 
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1 


1 


1 


1 



if center = 1 , do nothing, next kernel position 
if center = 0 {white} then test to see if should change to black... 
if sum of 8 neighbors >= 4, set center - 1 
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set image boundary to 0 

S Sobel edge operator, 3x3, (sum absolute values, amplitude only) 

Convolve input image with standard Sobel 3x3 (horizontal & vertical) 
kernels Ksh, Ksv producing inner products h & v respectively 



Ksh 
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Outputjj a abs(h) + abs(v) 
set image boundary to 0 
AD absolute deviation, 3x3 
3x3 kernel Kad 
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Calculate Mk & Sk on the kernel (8 elements) 

Mk = right shift 3 bits (sum Pij on Kad ) 

Sk = right shift 3 bits (sum abs(Pij - Mk) on Kad ) 

Outputjj = Sk 

set image boundary to 1 1 

AO region labeling, first pass 

2x2 kernel KaO over binary image per attached C-code segment 
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